This paper discusses recognition up to intensities of mix of primary facial expressions in real time. The proposed recognition method is compatible with the MPEG-4 high level expression Facial Animation Parameter (FAP). In our method, the whole facial image is considered as a single pattern without any block segmentation. As model features, an expression vector, viz. low global frequency coefficient (DCT) changes relative to neutral facial image of a person is used. These features are robust and good enough to deal with real time processing. To construct a person specific model, apex images of primary facial expression categories are utilized as references. Personal facial expression space (PFES) is constructed by using multidimensional scaling. PFES with its generalization capability maps an unknown input image relative to known reference images. As PFES possesses linear mapping characteristics, MPEG-4 high level expression FAP can be easily calculated by the location of the input face on PFES. Also, temporal variations of facial expressions can be seen on PFES as trajectories. Experimental results are shown to demonstrate the effectiveness of the proposed method.

Detection of facial emotions are mainly addressed by computer vision researchers based on facial display. Also detection of vocal expressions of emotions is found in research work done by acoustic researchers. Most of these research paradigms are devoted purely to visual or purely to auditory human emotion detection. However we found that it is very interesting to consider both of these auditory and visual informations together, for processing, since we hope this kind of multimodal information processing will become a datum of information processing in future multimedia era. By several intensive subjective evaluation studies we found that human beings recognize Anger, happiness, Surprise and Dislike by their visual appearance, compared to voice only detection. When the audio track of each emotion clip is dubbed with a different type of auditory emotional expression, still Anger, Happiness and Surprise were video dominant. However Dislike emotion gave mixed responses to different speakers. In both studies we found that Sadness and Fear emotions were audio dominant. As a conclusion to the paper we propose a method of facial emotion detection by using a hybrid approach, which uses multimodal informations for facial emotion recognition.